1. Unsupervised recurrent networks Barbara Hammer, Institute of Informatics, Clausthal University of Technology

2. Barbara Hammer Institut of Informatics 2 Unsupervised recursive networks Clausthal - Zellerfeld Brocken

3. Barbara Hammer Institut of Informatics 3 Unsupervised recursive networks

4. Barbara Hammer Institut of Informatics 4 Unsupervised recursive networks

5. Barbara Hammer Institut of Informatics 5 Unsupervised recursive networks

6. Barbara Hammer Institut of Informatics 6 Unsupervised recursive networks Prototype-based clustering …

7. Barbara Hammer Institut of Informatics 7 Unsupervised recursive networks Prototype based clustering  data contained in a real-vector space  prototypes characterized by locations in the data space  clustering induced by the receptive fields based on the euclidean metric

8. Barbara Hammer Institut of Informatics 8 Unsupervised recursive networks Vector quantization  init prototypes  repeat -present a data point -adapt the winner into the direction of the data point

9. Barbara Hammer Institut of Informatics 9 Unsupervised recursive networks Cost function  minimizes the cost function  online: stochastic gradient descent 

10. Barbara Hammer Institut of Informatics 10 Unsupervised recursive networks Neighborhood cooperation j=(j 1,j 2 ) wjwj Self-Organizing Map: regular lattice wjwj Neural gas: data optimum topology

11. Barbara Hammer Institut of Informatics 11 Unsupervised recursive networks Clustering recurrent data …

12. Barbara Hammer Institut of Informatics 12 Unsupervised recursive networks

13. Barbara Hammer Institut of Informatics 13 Unsupervised recursive networks Old models…

14. Barbara Hammer Institut of Informatics 14 Unsupervised recursive networks Old models Temporal Kohonen Map: x 1,x 2,x 3,x 4,…,x t, … d(x t,w i ) = |x t -w i | + α·d(x t-1,w i ) training: w i  x t Recurrent SOM: d(x t,w i ) = |y t | where y t = (x t -w i ) + α·y t-1 training: w i  y t

15. Barbara Hammer Institut of Informatics 15 Unsupervised recursive networks

16. Barbara Hammer Institut of Informatics 16 Unsupervised recursive networks Our model…

17. Barbara Hammer Institut of Informatics 17 Unsupervised recursive networks Merge neural gas/SOM x t,x t-1,x t-2,…,x 0 x t-1,x t- 2,…,x 0 CtCt (w,c) |x t – w| 2 xtxt |C t - c| 2 training: w  x t c  C t merge-context:: content of the winner

18. Barbara Hammer Institut of Informatics 18 Unsupervised recursive networks Merge neural gas/SOM  explicit context, global recurrence  w j : represents entry x t  c j : repesents the context which equals the winner content of the last time step  distance: d(x t,w j ) = α·|x t -w j | + (1-α)·|C t -c j | where C t = γ·w I(t-1) + (1-γ)·c I(t-1), I(t-1) winner in step t-1 (merge)  training w j  x t, c j  C t (w j,c j ) in ℝ nxn

19. Barbara Hammer Institut of Informatics 19 Unsupervised recursive networks Merge neural gas/SOM 42 50 33 45 32 42 41 40 34 39 33 38 40 37 35 36 34 35 Example: 42  33  33  34 C 1 = (42 + 50)/2 = 46C 2 = (33+45)/2 = 39 C 3 = (33+38)/2 = 35.5

20. Barbara Hammer Institut of Informatics 20 Unsupervised recursive networks Merge neural gas/SOM  speaker identification, japanese vovel ‘ae’ [UCI-KDD archive]  9 speakers, 30 articulations each 12-dim. cepstrum time MNG, 150 neurons: 2.7% test error MNG, 1000 neurons: 1.6% test error rule based: 5.9%, HMM: 3.8%

21. Barbara Hammer Institut of Informatics 21 Unsupervised recursive networks Merge neural gas/SOM Experiment:  classification of donor sites for C.elegans  5 settings with 10000 training data, 10000 test data, 50 nucleotides TCGA embedded in 3 dim, 38% donor [Sonnenburg, Rätsch et al.]  MNG with posterior labeling  512 neurons, γ=0.25, η=0.075, α: 0.999  [0.4,0.7]  14.06%±0.66% training error, 14.26%±0.39% test error  sparse representation: 512 · 6 dim

22. Barbara Hammer Institut of Informatics 22 Unsupervised recursive networks Merge neural gas/SOM Theorem – context representation: Assume  a map with merge context is given (no neighborhood)  a sequence x 0, x 1, x 2, x 3,… is given  enough neurons are available Then:  the optimum weight/context pair for x t is w = x t, c = ∑ i=0..t-1 γ(1-γ) t-i-1 ·x i  Hebbian training converges to this setting as a stable fixed point  Compare to TKM: -optimum weights are w = ∑ i=0..t (1-α) i ·x t-i / ∑ i=0..t (1-α) i -but: no fixed point for TKM  MSOM is the correct implementation of TKM

23. Barbara Hammer Institut of Informatics 23 Unsupervised recursive networks More models…

24. Barbara Hammer Institut of Informatics 24 Unsupervised recursive networks More models x t,x t-1,x t-2,…,x 0 x t-1,x t- 2,…,x 0 CtCt (w,c) |x t – w| 2 xtxt |C t - c| 2 training: w  x t c  C t Context: RSOM/TKM – neuron itself MSOM – winner content SOMSD – winner index RecSOM – all activations

25. Barbara Hammer Institut of Informatics 25 Unsupervised recursive networks More models TKMRSOMMSOMSOMSDRecSOM contextNeuron itself Winner content Winner indexActivation of all neurons encodingInput space Lattice spaceActivation space memorynN 2nN(d+n)N(N+n)N latticeall regular / hyperbolic all capacity<FSA FSA ≥ PDA* * for normalised WTA context

26. Barbara Hammer Institut of Informatics 26 Unsupervised recursive networks More models Experiment:  Mackey-Glass time series  100 neurons  different lattices  different contexts  evaluation by the temporal quantization error: average(mean activity k steps into the past - observed activity k steps into the past) 2

27. Barbara Hammer Institut of Informatics 27 Unsupervised recursive networks More models SOM RSOM NG RecSOM HSOMSD MNG SOMSD past now quantization error

28. Barbara Hammer Institut of Informatics 28 Unsupervised recursive networks So what?

29. Barbara Hammer Institut of Informatics 29 Unsupervised recursive networks So what?  inspection / clustering of high-dimensional events within their temporal context could be possible  strong regularization as for standard SOM / NG  possible training methods for reservoirs  some theory  some examples  no supervision  the representation of context is critical and not clear at all  training is critical and not clear at all

30. Barbara Hammer Institut of Informatics 30 Unsupervised recursive networks