An informal account of BackProp For each pattern in the training set: Compute the error at the output nodes Compute  w for each wt in 2 nd layer Compute delta (generalized error expression) for hidden units Compute  w for each wt in 1 st layer After amassing  w for all weights and, change each wt a little bit, as determined by the learning rate

Back-Propagation Algorithm We define the error term for a single node to be t i - y i xixi f yjyj w ij yiyi x i = ∑ j w ij y j y i = f(x i ) t i :target Sigmoid:

Backprop Details Here we go… Also refer to web notes for derivation

kji w jk w ij E = Error = ½ ∑ i (t i – y i ) 2 yiyi t i : target The derivative of the sigmoid is just The output layer learning rate

kji w jk w ij E = Error = ½ ∑ i (t i – y i ) 2 yiyi t i : target The hidden layer

Backprop learning algorithm n=1; initialize w(n) randomly; while (stopping criterion not satisfied and n<max_iterations) for each example (x,d) - run the network with input x and compute the output y - update the weights in backward order starting from those of the output layer: with computed using the (generalized) Delta rule end-for n = n+1; end-while;

Backpropagation Algorithm Initialize all weights to small random numbers For each training example do  For each hidden unit h:  For each output unit k:  For each hidden unit h:  Update each network weight w ij : with

What if all the input To hidden node weights are initially equal?

Momentum term The speed of learning is governed by the learning rate.  If the rate is low, convergence is slow  If the rate is too high, error oscillates without reaching minimum. Momentum tends to smooth small weight error fluctuations. the momentum accelerates the descent in steady downhill directions. the momentum has a stabilizing effect in directions that oscillate in time.

Convergence May get stuck in local minima Weights may diverge …but works well in practice Representation power:  2 layer networks : any continuous function  3 layer networks : any function

Local Minimum USE A RANDOM COMPONENT SIMULATED ANNEALING

Overfitting and generalization TOO MANY HIDDEN NODES TENDS TO OVERFIT

Overfitting in ANNs

Early Stopping (Important!!!) Stop training when error goes up on validation set

Stopping criteria Sensible stopping criteria:  total mean squared error change: Back-prop is considered to have converged when the absolute rate of change in the average squared error per epoch is sufficiently small (in the range [0.01, 0.1]).  generalization based criterion: After each epoch the NN is tested for generalization. If the generalization performance is adequate then stop. If this stopping criterion is used then the part of the training set used for testing the network generalization will not be used for updating the weights.

Architectural Considerations What is the right size network for a given job? How many hidden units? Too many: no generalization Too few: no solution Possible answer: Constructive algorithm, e.g. Cascade Correlation (Fahlman, & Lebiere 1990) etc

The number of layers and of neurons depend on the specific task. In practice this issue is solved by trial and error. Two types of adaptive algorithms can be used:  start from a large network and successively remove some nodes and links until network performance degrades.  begin with a small network and introduce new neurons until performance is satisfactory. Network Topology

Supervised vs Unsupervised Learning Backprop requires a 'target' how realistic is that? Hebbian learning is unsupervised, but limited in power How can we combine the power of backprop (and friends) with the ideal of unsupervised learning?

Autoassociative Networks input copy of input as target Network trained to reproduce the input at the output layer Non-trivial if number of hidden units is smaller than inputs/outputs Forced to develop compressed representations of the patterns Hidden unit representations may reveal natural kinds (e.g. Vowels vs Consonants) Problem of explicit teacher is circumvented

Problems and Networks Some problems have natural "good" solutions Solving a problem may be possible by providing the right armory of general-purpose tools, and recruiting them as needed Networks are general purpose tools. Choice of network type, training, architecture, etc greatly influences the chances of successfully solving a problem Tension: Tailoring tools for a specific job Vs Exploiting general purpose learning mechanism

Summary Multiple layer feed-forward networks  Replace Step with Sigmoid (differentiable) function  Learn weights by gradient descent on error function  Backpropagation algorithm for learning  Avoid overfitting by early stopping

ALVINN drives 70mph on highways

Use MLP Neural Networks when … (vectored) Real inputs, (vectored) real outputs You’re not interested in understanding how it works Long training times acceptable Short execution (prediction) times required Robust to noise in the dataset

Applications of FFNN Classification, pattern recognition: FFNN can be applied to tackle non-linearly separable learning problems.  Recognizing printed or handwritten characters,  Face recognition  Classification of loan applications into credit-worthy and non-credit-worthy groups  Analysis of sonar radar to determine the nature of the source of a signal Regression and forecasting: FFNN can be applied to learn non-linear functions (regression) and in particular functions whose inputs is a sequence of measurements over time (time series).

Extensions of Backprop Nets Recurrent Architectures Backprop through time

Elman Nets & Jordan Nets Updating the context as we receive input In Jordan nets we model “forgetting” as well The recurrent connections have fixed weights You can train these networks using good ol’ backprop Output Hidden ContextInput 1 α Output Hidden ContextInput 1

Recurrent Backprop we’ll pretend to step through the network one iteration at a time backprop as usual, but average equivalent weights (e.g. all 3 highlighted edges on the right are equivalent) abc unrolling 3 iterations abc abc abc w2 w1w3 w4 w1w2w3w4 abc

Connectionist Models in Cognitive Science Structured PDP (Elman) NeuralConceptualExistenceData Fitting Hybrid

5 levels of Neural Theory of Language Cognition and Language Computation Structured Connectionism Computational Neurobiology Biology MidtermQuiz Finals Neural Development Triangle Nodes Neural Net and learning Spatial Relation Motor Control Metaphor SHRUTI Grammar abstraction Pyscholinguistic experiments

The Color Story: A Bridge between Levels of NTL (

A Tour of the Visual System two regions of interest: –retina –LGN

The Physics of Light Light: Electromagnetic energy whose wavelength is between 400 nm and 700 nm. (1 nm = 10 meter) -6 © Stephen E. Palmer, 2002

The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red Yellow Blue Purple © Stephen E. Palmer, 2002

The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

Physiology of Color Vision Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision Two types of light-sensitive receptors

The Microscopic View

Rods and Cones in the Retina

What Rods and Cones Detect Notice how they aren’t distributed evenly, and the rod is more sensitive to shorter wavelengths

© Stephen E. Palmer, 2002 Three kinds of cones: Absorption spectra Implementation of Trichromatic theory Physiology of Color Vision Opponent Processes: R/G = L-M G/R = M-L B/Y = S-(M+L) Y/B = (M+L)-S

© Stephen E. Palmer, 2002 Double Opponent Cells in V1 Physiology of Color Vision G+R-G+R- G+R-G+R- R+G-R+G- R+G-R+G- Red/Green Y+B-Y+B- Y+B-Y+B- B+Y-B+Y- B+Y-B+Y- Blue/Yellow

Color Blindness Not everybody perceives colors in the same way! What numbers do you see in these displays? © Stephen E. Palmer, 2002

Theories of Color Vision A Dual Process Wiring Diagram © Stephen E. Palmer, 2002 Trichromatic Stage Opponent Process Stage

Color Naming © Stephen E. Palmer, 2002 Basic Color Terms (Berlin & Kay) Criteria: 1. Single words -- not “light-blue” or “blue-green” 2. Frequently used -- not “mauve” or “cyan” 3. Refer primarily to colors -- not “lime” or “gold” 4. Apply to any object -- not “roan” or “blond”

Color Naming © Stephen E. Palmer, 2002 BCTs in English Red Green Blue Yellow Black White Gray Brown Purple Orange* Pink

Color Naming © Stephen E. Palmer, 2002 Five more BCTs in a study of 98 languages Light-Blue Warm Cool Light-Warm Dark-Cool

The WCS Color Chips Basic color terms: –Single word (not blue-green) –Frequently used (not mauve) –Refers primarily to colors (not lime) –Applies to any object (not blonde) FYI: English has 11 basic color terms

Results of Kay’s Color Study If you group languages into the number of basic color terms they have, as the number of color terms increases, additional terms specify focal colors Stage IIIIIIa / IIIbIVVVIVII W or R or YWWWWWW Bk or G or BuR or Y RRRR Bk or G or BuG or BuYYYY BkG or BuGGG BkBu WBk RY+Bk (Brown) YR+W (Pink) Bk or G or BuR + Bu (Purple) R+Y (Orange) B+W (Grey)

Color Naming © Stephen E. Palmer, 2002 Typical “developmental” sequence of BCTs

Color Naming © Stephen E. Palmer, 2002 Studied color categories in two ways Boundaries Best examples (Berlin & Kay)

Color Naming © Stephen E. Palmer, 2002 MEMORY : Focal colors are remembered better than nonfocal colors. LEARNING: New color categories centered on focal colors are learned faster. Categorization: Focal colors are categorized more quickly than nonfocal colors. (Rosch)

Color Naming Degree of Membership Fuzzy set theory (Zadeh) A fuzzy logical model of color naming (Kay & Mc Daniel) © Stephen E. Palmer, 2002

Color Naming © Stephen E. Palmer, 2002 “Primary” color categories

Color Naming © Stephen E. Palmer, 2002 “Primary” color categories Red Green Blue Yellow Black White

Color Naming © Stephen E. Palmer, 2002 “Derived” color categories Fuzzy logical “AND f ”

Color Naming © Stephen E. Palmer, 2002 “Derived” color categories Orange = Red AND f Yellow Purple = Red AND f Blue Gray = Black AND f White Pink = Red AND f White Brown = Yellow AND f Black (Goluboi = Blue AND f White)

Color Naming © Stephen E. Palmer, 2002 “Composite” color categories Fuzzy logical “OR f ” Warm = Red Or f Yellow Cool = Blue Or f Green Light-warm = White Or f Warm Dark-cool = Black Or f Cool

Color Naming © Stephen E. Palmer, 2002