CS 621 Artificial Intelligence Lecture 29 – 22/10/05 Prof. Pushpak Bhattacharyya SOM, Theoretical Aspects Of Machine Learning, Probably Approximately Correct Learning.

A … P neurons o/p layer Wp … … . n neurons Clusters: A : A B : : C :

Wp(n+1) = Wp(n) + η (I(n) – Wp(n)) Competitive learning: Weights to the winner neuron only are changed.

Generating function application to recurrence relation solving. Example: Max. No of regions on 2-Dim plane produced by n lines.

L1 L3 y L2 R1 = 2 R2 = R1 + 2 = 2+ 2 = 4 L4 R3 = R2 + 3 = 4 + 3 = 7 R4 = 7 + 4 = 11 Rn = Rn-1 + n R1 = 2 x

Rn = Rn-1 + n Rn-1 = Rn-2 + (n-1) Rn-2 = Rn-3 + (n-2) . R2 = R1 + 2

Rn = n + (n-1) + (n-2)+ …..+R1 + 2 = n+(n-1)+(n-2)+…..+2+2 =n(n+1)/2 + 1

Generating Function Method Key idea: Rns are generated by infinite series function ,i.e, F(x) = R1 + R2x + R3x2 + ………………+ Rnxn-1+ ….. ∞ Rn = Coeff of xn-1

Rn = Rn-1 + n Rn - Rn-1 = n R2 - R1 = 2 R3 – R2 = 3 . (Continued) Rn = Rn-1 + n Rn - Rn-1 = n R2 - R1 = 2 R3 – R2 = 3 .

(Continued) F(x) = R1 + R2x + R3x2 + ………………+ Rnxn-1+ ….. ∞ (1) xF(x) = R1x + R2x2 + R3x3 + ……………+ Rnxn+ ….. ∞ (2) - 2 gives (1-x) F(x) = R1 + (R2 – R1)x + (R3 – R2)x2 + ……………+ (Rn – Rn-1)xn-1+ ….. ∞

(Continued) (1-x) F(x) = 2 + 2x + 3x2 + ……………+ nxn-1+ ….. ∞ F(x) = (1-x)-1(2 + 2x + 3x2 + ……………+ nxn-1+ ….. ∞) = ( 1 + x + x2 + x3 +…+ xn-1+…..) (2 + 2x + 3x2 + 4x3 +…+ nxn-1+…..)

Co eff of Xn-1 from the expression for f(x) (Continued) Co eff of Xn-1 from the expression for f(x) = ( n + n-1 + n-2 + … + 2+ 2 ) n(n+1)/ 2 + 1

Clustering Algos 1. Competitive learning 2. K – means clustering 3. Self organization / Kohonen net 4. Counter Propagation

Two packages 1. SNNS – Stuttgart Neural Net Simulator 2. WEKA Package 3. MATLAB

Data to train IRIS data: 3 - class data X- OR, parity, Majority

IRIS Data Classification of flowers based on four attributes in 3 classes c1, c2 and C2 1 Petal length A1 2 Petal width A2 3 Sepal length A3 4 Sepal width A4

Google on “Machine learning Data” IRIS 120 – data points < A1, A2, A3, A4 >1 --- C1 < A1, A2, A3, A4 >2 --- C3 ……….

C1, C2, C3 each has 50 data points in it 80 – 20 rule for ML Training – 40 and Testing -- 10

What you have to do for IRIS choose 40 points randomly from C1, C2 and C3 1. Run BP on FF n/w 2. Run SOM assuming unlabelled data. See if you can discover 3 cluster

Two part assignment Supervised Hidden layer to be found 4 neurons for 4 attributes

Very good performance on test data should be achieved. BP for IRIS error 1% Training iterations EPOCHS Very good performance on test data should be achieved.

Cluster Discovery By SOM/Kohenen Net 4 I/p neurons A1 A2 A3 A4

K – means clustering K o/p neurons are required from the knowledge of k clusters being present. …… 26 neurons Full connection …… n neurons

Steps 1. Initialize the weights randomly. 2. Ik is the vector presented at kth iteration. 3. Find W* such that |w* - Ik| < |wj - Ik| for all j

4. make W*(new) = W* (old) + η(Ik - w* ). 5. k  K +1 ; go to 3. 6. Go to 2 until the error is below a threshold.

K means 1. Initialize weights randomly. 2. Ik in the I/p vector presented at kth. step of one iteration over all the training data (EPOCH). 3. Find the W* such that |w* - Ik| is minimum. 4. Find W* for Ik+1..

5. See the partitions at the end of the epoch. 6. Find centroid of each partition. 7. Make the weight equal to the centroid of the partition. 8. Take next epoch.

9. Keep doing until the error is below a threshold. Key idea form competitive learning: Weight change incorporate after every epoch.