2. An informal account of BackProp For each pattern in the training set: Compute the error at the output nodes Compute  w for each wt in 2 nd layer Compute delta (generalized error expression) for hidden units Compute  w for each wt in 1 st layer After amassing  w for all weights and, change each wt a little bit, as determined by the learning rate

3. Back-Propagation Algorithm We define the error term for a single node to be t i - y i xixi f yjyj w ij yiyi x i = ∑ j w ij y j y i = f(x i ) t i :target Sigmoid:

4. Backprop Details Here we go… Also refer to web notes for derivation

5. kji w jk w ij E = Error = ½ ∑ i (t i – y i ) 2 yiyi t i : target The derivative of the sigmoid is just The output layer learning rate

6. kji w jk w ij E = Error = ½ ∑ i (t i – y i ) 2 yiyi t i : target The hidden layer

7. Backprop learning algorithm n=1; initialize w(n) randomly; while (stopping criterion not satisfied and n<max_iterations) for each example (x,d) - run the network with input x and compute the output y - update the weights in backward order starting from those of the output layer: with computed using the (generalized) Delta rule end-for n = n+1; end-while;

8. Backpropagation Algorithm Initialize all weights to small random numbers For each training example do  For each hidden unit h:  For each output unit k:  For each hidden unit h:  Update each network weight w ij : with

9. What if all the input To hidden node weights are initially equal?

10. Momentum term The speed of learning is governed by the learning rate.  If the rate is low, convergence is slow  If the rate is too high, error oscillates without reaching minimum. Momentum tends to smooth small weight error fluctuations. the momentum accelerates the descent in steady downhill directions. the momentum has a stabilizing effect in directions that oscillate in time.

11. Convergence May get stuck in local minima Weights may diverge …but works well in practice Representation power:  2 layer networks : any continuous function  3 layer networks : any function

12. Local Minimum USE A RANDOM COMPONENT SIMULATED ANNEALING

13. Overfitting and generalization TOO MANY HIDDEN NODES TENDS TO OVERFIT

14. Overfitting in ANNs

15. Early Stopping (Important!!!) Stop training when error goes up on validation set

16. Stopping criteria Sensible stopping criteria:  total mean squared error change: Back-prop is considered to have converged when the absolute rate of change in the average squared error per epoch is sufficiently small (in the range [0.01, 0.1]).  generalization based criterion: After each epoch the NN is tested for generalization. If the generalization performance is adequate then stop. If this stopping criterion is used then the part of the training set used for testing the network generalization will not be used for updating the weights.

17. Architectural Considerations What is the right size network for a given job? How many hidden units? Too many: no generalization Too few: no solution Possible answer: Constructive algorithm, e.g. Cascade Correlation (Fahlman, & Lebiere 1990) etc

18. The number of layers and of neurons depend on the specific task. In practice this issue is solved by trial and error. Two types of adaptive algorithms can be used:  start from a large network and successively remove some nodes and links until network performance degrades.  begin with a small network and introduce new neurons until performance is satisfactory. Network Topology

19. Supervised vs Unsupervised Learning Backprop requires a 'target' how realistic is that? Hebbian learning is unsupervised, but limited in power How can we combine the power of backprop (and friends) with the ideal of unsupervised learning?

20. Autoassociative Networks input copy of input as target Network trained to reproduce the input at the output layer Non-trivial if number of hidden units is smaller than inputs/outputs Forced to develop compressed representations of the patterns Hidden unit representations may reveal natural kinds (e.g. Vowels vs Consonants) Problem of explicit teacher is circumvented

21. Problems and Networks Some problems have natural "good" solutions Solving a problem may be possible by providing the right armory of general-purpose tools, and recruiting them as needed Networks are general purpose tools. Choice of network type, training, architecture, etc greatly influences the chances of successfully solving a problem Tension: Tailoring tools for a specific job Vs Exploiting general purpose learning mechanism

22. Summary Multiple layer feed-forward networks  Replace Step with Sigmoid (differentiable) function  Learn weights by gradient descent on error function  Backpropagation algorithm for learning  Avoid overfitting by early stopping

23. ALVINN drives 70mph on highways

24. Use MLP Neural Networks when … (vectored) Real inputs, (vectored) real outputs You’re not interested in understanding how it works Long training times acceptable Short execution (prediction) times required Robust to noise in the dataset

25. Applications of FFNN Classification, pattern recognition: FFNN can be applied to tackle non-linearly separable learning problems.  Recognizing printed or handwritten characters,  Face recognition  Classification of loan applications into credit-worthy and non-credit-worthy groups  Analysis of sonar radar to determine the nature of the source of a signal Regression and forecasting: FFNN can be applied to learn non-linear functions (regression) and in particular functions whose inputs is a sequence of measurements over time (time series).

27. Extensions of Backprop Nets Recurrent Architectures Backprop through time

28. Elman Nets & Jordan Nets Updating the context as we receive input In Jordan nets we model “forgetting” as well The recurrent connections have fixed weights You can train these networks using good ol’ backprop Output Hidden ContextInput 1 α Output Hidden ContextInput 1

29. Recurrent Backprop we’ll pretend to step through the network one iteration at a time backprop as usual, but average equivalent weights (e.g. all 3 highlighted edges on the right are equivalent) abc unrolling 3 iterations abc abc abc w2 w1w3 w4 w1w2w3w4 abc

30. Connectionist Models in Cognitive Science Structured PDP (Elman) NeuralConceptualExistenceData Fitting Hybrid

31. 5 levels of Neural Theory of Language Cognition and Language Computation Structured Connectionism Computational Neurobiology Biology MidtermQuiz Finals Neural Development Triangle Nodes Neural Net and learning Spatial Relation Motor Control Metaphor SHRUTI Grammar abstraction Pyscholinguistic experiments

32. The Color Story: A Bridge between Levels of NTL (http://www.ritsumei.ac.jp/~akitaoka/color-e.html

33. A Tour of the Visual System two regions of interest: –retina –LGN

34. The Physics of Light Light: Electromagnetic energy whose wavelength is between 400 nm and 700 nm. (1 nm = 10 meter) -6 © Stephen E. Palmer, 2002

35. The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

36. The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 © Stephen E. Palmer, 2002

37. The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

38. The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

39. The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

40. The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

41. Physiology of Color Vision Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision Two types of light-sensitive receptors

42. The Microscopic View

43. http://www.iit.edu/~npr/DrJennifer/visual/retina.html Rods and Cones in the Retina

44. What Rods and Cones Detect Notice how they aren’t distributed evenly, and the rod is more sensitive to shorter wavelengths

45. © Stephen E. Palmer, 2002 Three kinds of cones: Absorption spectra Implementation of Trichromatic theory Physiology of Color Vision Opponent Processes: R/G = L-M G/R = M-L B/Y = S-(M+L) Y/B = (M+L)-S

46. © Stephen E. Palmer, 2002 Double Opponent Cells in V1 Physiology of Color Vision G+R-G+R- G+R-G+R- R+G-R+G- R+G-R+G- Red/Green Y+B-Y+B- Y+B-Y+B- B+Y-B+Y- B+Y-B+Y- Blue/Yellow

47. Color Blindness Not everybody perceives colors in the same way! What numbers do you see in these displays? © Stephen E. Palmer, 2002

48. Theories of Color Vision A Dual Process Wiring Diagram © Stephen E. Palmer, 2002 Trichromatic Stage Opponent Process Stage

49. Color Naming © Stephen E. Palmer, 2002 Basic Color Terms (Berlin & Kay) Criteria: 1. Single words -- not “light-blue” or “blue-green” 2. Frequently used -- not “mauve” or “cyan” 3. Refer primarily to colors -- not “lime” or “gold” 4. Apply to any object -- not “roan” or “blond”

50. Color Naming © Stephen E. Palmer, 2002 BCTs in English Red Green Blue Yellow Black White Gray Brown Purple Orange* Pink

51. Color Naming © Stephen E. Palmer, 2002 Five more BCTs in a study of 98 languages Light-Blue Warm Cool Light-Warm Dark-Cool

52. The WCS Color Chips Basic color terms: –Single word (not blue-green) –Frequently used (not mauve) –Refers primarily to colors (not lime) –Applies to any object (not blonde) FYI: English has 11 basic color terms

53. Results of Kay’s Color Study If you group languages into the number of basic color terms they have, as the number of color terms increases, additional terms specify focal colors Stage IIIIIIa / IIIbIVVVIVII W or R or YWWWWWW Bk or G or BuR or Y RRRR Bk or G or BuG or BuYYYY BkG or BuGGG BkBu WBk RY+Bk (Brown) YR+W (Pink) Bk or G or BuR + Bu (Purple) R+Y (Orange) B+W (Grey)

54. Color Naming © Stephen E. Palmer, 2002 Typical “developmental” sequence of BCTs

55. Color Naming © Stephen E. Palmer, 2002 Studied color categories in two ways Boundaries Best examples (Berlin & Kay)

56. Color Naming © Stephen E. Palmer, 2002 MEMORY : Focal colors are remembered better than nonfocal colors. LEARNING: New color categories centered on focal colors are learned faster. Categorization: Focal colors are categorized more quickly than nonfocal colors. (Rosch)

57. Color Naming Degree of Membership Fuzzy set theory (Zadeh) A fuzzy logical model of color naming (Kay & Mc Daniel) © Stephen E. Palmer, 2002

58. Color Naming © Stephen E. Palmer, 2002 “Primary” color categories

59. Color Naming © Stephen E. Palmer, 2002 “Primary” color categories Red Green Blue Yellow Black White

60. Color Naming © Stephen E. Palmer, 2002 “Derived” color categories Fuzzy logical “AND f ”

61. Color Naming © Stephen E. Palmer, 2002 “Derived” color categories Orange = Red AND f Yellow Purple = Red AND f Blue Gray = Black AND f White Pink = Red AND f White Brown = Yellow AND f Black (Goluboi = Blue AND f White)

62. Color Naming © Stephen E. Palmer, 2002 “Composite” color categories Fuzzy logical “OR f ” Warm = Red Or f Yellow Cool = Blue Or f Green Light-warm = White Or f Warm Dark-cool = Black Or f Cool

63. Color Naming © Stephen E. Palmer, 2002