1. Machine Learning
12. Local Models

2. Introduction Clustering Use neural network for unsupervised learning
Using K-means algorithm
Iterative algorithm
Batch learning
Use neural network for unsupervised learning
Online learning
Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

3. Online K-means Reconstruction Error
For batch k-means, center update is
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

4. Online k-means Reconstruction error for single instance
Using gradient descent
Move closest center to the direction of new sample
Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

5. Competitive Learning
Based on 2004 Introduction to Machine Learning © The MIT Press (V1.1)

6. Competitive Learning
Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

7. Using Dot product If All centers have the same norm
Minimum Euclidian distance
Correspond to maximum dot product
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

8. Choosing maximum using NN
Using Recurrent Networks
Lateral inhibition
Positive(excitatory) recurrent connection to itself
Negative(inhibitory) recurrent connection to its neighbors
Suitable weights and activation function converge to a maximum
Singe output is 1
All other 0
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

9. NN network Winner-take-all network
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

10. Hebbian learning Update
First term: increase weight if input and output are activated together (correlated)
Second term: prevent unbounded weight growth
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

11. Adaptive Resonance Theory
Incremental; add a new cluster if not covered;
vigilance, ρ
(Carpenter and Grossberg, 1988)
Based on Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

12. Self-Organizing Maps
Units have a neighborhood defined; mi is “between” mi-1 and mi+1, and are all updated together
One-dim map:
(Kohonen, 1990)
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

13. Locally Receptive Units
Divide the input space into local regions and learn simple (e.g. constant/linear) models in each patch
Radial-basis func, mixture of experts
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

14. Local vs Distributed Representation
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

15. Radial-Basis Functions
Locally-tuned units:
Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

16. Training RBF Hybrid learning: Fully supervised RBF is differentiable
First layer centers and spreads:
Unsupervised k-means
Second layer weights: Supervised gradient-descent
Fully supervised
RBF is differentiable
Back propagation
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

17. Rule-Based Knowledge
Incorporation of prior knowledge (before training)
Rule extraction (after training) (Tresp et al., 1997)
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)