1 Analysis of Ensemble Learning using Simple Perceptrons based on Online Learning Theory Seiji MIYOSHI 1 Kazuyuki HARA 2 Masato OKADA 3,4,5 1 Kobe City College of Tech., 2 Tokyo Metropolitan College of Tech., 3 University of Tokyo 4 RIKEN BSI, 5 Intelligent Cooperation and Control, PRESTO, JST

2 ABSTRACT Ensemble learning of K simple perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. One purpose of statistical learning theory is to theoretically obtain the generalization error. We show that ensemble generalization error can be calculated by using two order parameters, that is, the similarity between a teacher and a student, and the similarity among students. The differential equations that describe the dynamical behaviors of these order parameters are derived in the case of general learning rules. The concrete forms of these differential equations are derived analytically in the cases of three well-known rules: Hebbian learning, perceptron learning and AdaTron learning. Ensemble generalization errors of these three rules are calculated by using the results determined by solving their differential equations. As a result, these three rules show different characteristics in their affinity for ensemble learning, that is “maintaining variety among students”. Results show that AdaTron learning is superior to the other two rules with respect to that affinity. ‘

3 BACKGROUND Ensemble learning has recently attracted the attention of many researchers. Ensemble learning means to combine many rules or learning machines (students in the following) that perform poorly. Theoretical studies analyzing the generalization performance by using statistical mechanics have been performed vigorously. Hara and Okada theoretically analyzed the case in which students are linear perceptrons. Hebbian learning, perceptron learning and AdaTron learning are well-known as learning rules for a nonlinear perceptron, which decides its output by sign function. Determining differences among ensemble learnings with Hebbian learning, perceptron learning and AdaTron learning, is a very attractive problem, but it is one that has never been analyzed. OBJECTIVE We discuss ensemble learning of K simple perceptrons within the framework of online learning and finite K.

4 MODEL Common input x to teacher and all students in the same order. Input x, once used for an update, is abandoned. (Online learning) Update of student is independent each other. Two methods are treated to decide an ensemble output. One is the majority vote (MV) of students, and the other is the weight mean (WM). length of student Input: Teacher ： Student: TeacherStudents 12K

5 Generalization Error ε g : Probability that an ensemble output disagrees with that of the teacher for a new input x THEORY Similarity between teacher and student Similarity among students

6 Differential equations describing l and R (known result) Differential equation describing q (new result)

7 Hebbian (known result) (new result) RESULTS

8 Perceptron (known result) (new result)

9 AdaTron (known result) (new result)

10 Generalization Error Hebbian Perceptron AdaTron K= ∞ K=1

11 Similarity between teacher and student Similarity among students DISCUSSION

12 To maintain the variety of students is important in ensemble learning. →Relationship between R and q is essential. B J k J k' B J k J q kk' q is small → Effect of ensemble is strong. q is large → Effect of ensemble is small.

13 Dynamical behaviors of R and q Hebbian Perceptron AdaTron Relationship between R and q