Fourth International Symposium on Neural Networks (ISNN) June 3-7, 2007, Nanjing, China A Hierarchical Self-organizing Associative Memory for Machine Learning Janusz A. Starzyk, Ohio University Haibo He, Stevens Institute of Technology Yue Li, O2 Micro Inc

2/23 Outline  Introduction;  Associative learning algorithm;  Memory network architecture and operation;  Simulation analysis;  Conclusion and future research;

3/23 Introduction: A biological point of view Source: “The computational brain” by P. S. Churchland and T. J. Sejnowski Memory is a critical component for understanding and developing natural intelligent machines/systems The question is: How???

4/23 Introduction: self-organizing learning array (SOLAR) Characteristics: * Self-organization * Sparse and local interconnections * Dynamically reconfigurable * Online data-driven learning Other Neurons Nearest neighbour neuron Remote neurons System clock ID: information deficiency II: information index

5/23 Introduction: from SOLAR to AM Characteristics:  Self-organization;  Sparse and local interconnections;  Feedback propagation;  Information inference;  Hierarchical organization;  Robust and self-adaptive;  Capable of both hetero-associative (HA) and auto-associative (AA) Feed forward only Feed forward Feed backward

6/23 Outline  Introduction; Associative learning algorithm;  Memory network architecture and operation;  Simulation analysis;  Conclusion and future research;

7/23 Basic learning element Self-determination of the function value: An example:

8/23 Signal strength (SS) Signal strength (SS) =| Signal value – logic threshold| (SS range: [0, 1])  Provides a coherent way to determine when to trigger an association;  Helps to resolve multiple feedback signals;

9/23 Three types of associations  IOA: Input only association;  OOA: Output only association;  INOUA: Input-output association;

10/23 Probability based associative learning algorithm  Case 1: Given the values of both inputs, decide the output value;

11/23 Probability based associative learning algorithm  Case 2: Given the values of one input and an un-defined output, decide the value of the other input; For instance:

12/23 Probability based associative learning algorithm  Case 3: Given the values of the output, decide the values of both inputs;

13/23 Probability based associative learning algorithm  Case 4: Given the values of one input and the output, decide the other input value; For instance:

14/23 Outline  Introduction;  Associative learning algorithm; Memory network architecture and operation;  Simulation analysis;  Conclusion and future research;

15/23 Network operations Feedback operationFeed forward operation Depth Input data Depth ?.??.?

16/23 Memory operation Undefined signal Defined signal Recovered signal Input data Signal resolved based on SS

17/23 Outline  Introduction;  Associative learning algorithm;  Memory network architecture and operation; Simulation analysis;  Conclusion and future research;

18/23 Hetero-associative memory: Iris database classification N-bits sliding-bar coding mechanism: Features: Class identity labels: In our simulation: N=80, L=20, M=30 3 classes, 4 numeric attributes, 150 instances

19/23 Neuron association pathway Classification accuracy: 96%

20/23 Auto-associative memory: Panda image recovery 30% missing pixels Original image 64x64 binary image Error: % Block half Error: 2.42% 64 x 64 binary panda image: for a black pixel; for a white pixel;

21/23 Outline  Introduction;  Associative learning algorithm;  Memory network architecture and operation;  Simulation analysis; Conclusion and future research;

22/23 Conclusion and future research  Hierarchical associative memory architecture;  Probabilistic information processing, transmission, association and prediction;  Self-organization;  Self-adaptive;  Robustness;

23/23 It’s all about design natural intelligent machines ! Future research  Multiple-inputs (>2) association mechanism;  Dynamically self-reconfigurable;  Hardware implementation;  Facilitate goal-driven learning;  Spatio-temporal memory organization; How far are we??? “Brain On Silicon” will not just be a dream or scientific fiction in the future! 3DANN Picture source: and Irvine Sensors Corporation (Costa Mesa, CA)