A Hybrid Self-Organizing Neural Gas Network James Graham and Janusz Starzyk School of EECS, Ohio University Stocker Center, Athens, OH USA IEEE World Conference on Computational Intelligence (WCCI’08) June 1-6, 2008 Hong Kong

Introduction Self Organizing Networks – Useful for representation building in unsupervised learning – Useful for clustering, visualization and feature maps – Numerous applications in surveillance, traffic monitoring, flight control, rescue mission, reinforcement learning, etc. Some Types of Self Organizing Networks – Traditional Self-Organizing Map – Parameterless SOM – Neural Gas Network – Growing Neural Gas – Self-Organizing Neural Gas (SONG)

Description of the approach - Fritzke’s GNG Network Algorithm Highlights 1) GNG starts with a set A of two units a and b at random positions w a and w b in R n 2) In the set A find two nearest neighbors s 1 and s 2 to the input signal x. 3) Connect s 1 and s 2, with an edge and set the edge age to zero. 4) Adjust the positions of s 1 and its neighborhood by a constant times (x-s 1 ). (  b for s 1 and  n for the neighborhood) 5) Remove edges in the neighborhood that are older than a max. 6) Place a new node every λ cycles between the node with greatest error and its nearest neighbor. 7) Reduce error of the node with the maximum error and its nearest neighbor by  %, and add the removed error to the new node. 8) Reduce error of all nodes by a constant (  ) times their current error.

Example Example of Fritzke’s network results for 40,000 iterations with the following constants:  b =0.05,  n =.0006, a max =88, =200,  =.5,  =

Description of the approach - Proposed Hybrid SONG Network Algorithm Highlights 1) SONG starts with a random pre-generated network of a fixed size. 2) Connections get “stiffer” with age, making their weight harder to change. 3) Error is calculated after the node position updates rather than before. 4) Weight adjustment and error distribution are functions of a distance rather than arbitrary, hard to set constants. 5) Edge connections are removed only under the following conditions: – When a connection is added and the node has a long connection 2x greater than its average connection length - the long edge is removed. – When a node is moved and has at least 2 connections (after attaching to its destination node) - its longest connection is removed.

Description of the approach - Modification of new data neighborhood “Force” calculations |w s -  | |w N -  | |w 2 -  | |w 1 -  | s  nearest neighbor new data remove connection to a distant neighbor >2 mean node removed if orphaned Weight adjustment Error increase Age increase by 1

Description of the approach - Node replacement Select a node with the minimum error E sk Spread E sk to its s k neighborhood sqsq maximum error node minimum error node moved sksk

Description of the approach - Node replacement Select a node with the minimum error E sk Spread E sk to its s k neighborhood sksk sqsq maximum error node longest connection removed Insert s k to the neighborhood of s q using weights Remove the longest connection Spread half of s q neighborhood error to s k

Results Initial network structure with 1 random connection per node (for 200 nodes)

Results (cont.) Structure resulting form 1 initial random connection.

Results (cont.) Connection equilibrium reached for 1 initial connection.

Results (cont.) Structure resulting from 16 initial random connections.

Connection equilibrium for 16 initial connections. Results (cont.)

Video of Network Progression Hybrid SONG NetworkFritzke GNG Network

Results (cont.) 2-D comparison, with SOM network Salient features of the SOM algorithm: – The SOM network starts as a predefined grid and is adjusted over many iterations. – Connections are fixed and nodes are not inserted, moved, or relocated out of their preexisting grid. – Weight adjustments occur over the entire grid and are controlled by weighted distance to the data point.

Growing SONG Network Number of nodes in SONG can be automatically obtained The SONG network starts with a few randomly placed nodes and build itself up until an equilibrium is reached between the network size and the error. A node is added every λ cycles if MaxError > AveError + Constant Equilibrium appears to be ~200 nodes.

Growing SONG Network (cont.) Error handling in growing SONG network was modified. The error is “reset” and recomputed after the equilibrium was reached Network continues to learn reaching new equilibrium Approximation accuracy vary from run to run

Growing SONG Network (cont.) The results of growing SONG network run (on the right) compared to the simpler static approach (on the left).

Other Applications - Sparsely connected hierarchical sensory network The major features of the SONG algorithm such as the weight adjustment, error calculation, and neighborhood selection are utilized in building self- organizing sparsely connected hierarchical networks. The sparse hierarchical network is locally connected based on neurons’ firing correlation Feedback and time based correlation are used for invariant object recognition.

Other Applications - Sparsely connected hierarchical sensory network (cont.) Correlation/PDF Example Wiring area

Other Applications - Sparsely connected hierarchical network (cont.) Correlation based wiring Sparse hierarchical representations Declining neurons’ activations

Conclusions The SONG algorithm is more biologically plausible than Fritzke’s GNG algorithm. Specifically: – Weight and error adjustment are not parameter based. – Connections become stiffer with age rather than being removed at a maximum age as in Fritzke’s method. – Network has all neurons from the beginning SONG approximates data distribution faster than the other methods tested. Connectivity between neurons is automatically obtained and depends on the parameter that controls edge removal and the network size. The number of neurons can be automatically obtained in growing SONG to achieve the desired accuracy.

Future Work Adapt the SONG algorithm to large input spaces (high dimensionality, i.e. images) Adapt the SONG algorithm to a hierarchical network. – Possible applications in feature extraction, representation building, and shape recognition. Insert new nodes as needed to reduce error. Optimize the network design.

Questions ?