1. Lecture 12

2. Outline of Rule-Based Classification 1. Overview of ANN 2. Basic Feedforward ANN 3. Linear Perceptron Algorithm 4. Nonlinear and Multilayer Perceptron 5. Advanced ANN

3. 1. Overview of ANNs Inspired by neuroscience of the brain Neurons linked together by axons (strands of fiber) Axons transmit nerve impulses between neurons Dendrites connect neurons to axons of other neurons at synapses Learning happens through changes in synaptic connection strength

4. 1. Artificial Neural Networks

5. Artificial Neural Networks (ANNs) Perceptron Invented at Cornell Aeronautical Laboratory in 1957 by Frank Rosenblatt Single layer feed-forward neural network Initially promising but ultimately disappointing – only able to learn linearly separable patterns Minsky and Papert extended to multi-layer perceptrons

6. Artificial Neural Networks (ANNs) Basic types of ANNs Feedforward No directed cycles Multilayer perceptron Recurrent Directed cycles Often used for handwriting recognition

7. 2. Example of Basic ANN Output Y is 1 if at least two of the three inputs are equal to 1.

8. Artificial Neural Networks (ANN)

9. Model is an assembly of inter-connected nodes and weighted links Output node sums up each of its input value according to the weights of its links Compare output node against some threshold t Perceptron Model or

10. 3. Perceptron of ANNs Algorithm for learning binary classifier Function that maps input x to output f(x) given by

11. 3. Perceptron Decision Boundary

12. 3. Perceptron of ANNs

13. Algorithm for learning ANN Initialize the weights (w 0, w 1, …, w k ) Adjust the weights in such a way that the output of ANN is consistent with class labels of training examples Objective function: Find the weights w i ’s that minimize the above objective function e.g., backpropagation algorithm

14. 3. Perceptron Learning Algorithm

15. 4. No linear Hyperplane boudary

16. Multilayer ANN Training ANN means learning the weights of the neurons

17. 4. Types of Activation functions

18. 4. Two Layer ANN for the XOR

19. 4. Type of error functions

21. 4. Advanced ANNs Recurrent ANN examples Fully recurrent Long short term memory (Jürgen Schmidhuber)

22. Artificial Neural Networks (ANNs) Recurrent ANN examples Hopfield (ECANs) Symmetric connections John Hopfield, 1982 Attractor network: dynamics guaranteed to converge Can function as associative memory

23. Artificial Neural Networks (ANNs) Deep learning architectures Hierarchical temporal memory (Jeff Hawkins and Dileep George) Deep belief networks (George Hinton) Convolutional networks (Yann Lecun, Yoshua Bengio) Deep Spatiotemporal Inference Networks (Itamar Arel) Google Deepmind

24. Artificial Neural Networks (ANNs) Basic learning mechanisms Supervised learning Infer mapping implied by the training data Gradient descent/Backpropagation

25. Artificial Neural Networks (ANNs) Basic learning mechanisms Unsupervised learning Minimize some given cost/energy function Reinforcement learning Data generated by agent’s interactions with environment Agent observes accumulated costs and adjust actions accordingly

26. Artificial Neural Networks (ANNs) Characteristics of ANNs Choice of model Depends upon application Complex models generally more difficult to learn Learning algorithm May require considerable experimentation to determine appropriate cost function and parameters

27. Artificial Neural Networks (ANNs) Characteristics of ANNs Choice of threshold function ANNs can be robust Easily implemented in parallel Neuromorphic computing (IBM)

28. Relevant Reference Books Gödel, Escher, Bach – an Eternal Golden Braid By Douglas R. Hofstadter, 1999. Pulitzer Prize Winner