Neural Networks

Introduction Artificial Neural Networks (ANN) Connectionist computation Parallel distributed processing Biologically Inspired computational models Machine Learning Artificial intelligence "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success.

History McCulloch and Pitts introduced the Perceptron in 1943. Simplified model of a biological neuron The drawback in the late 1960's (Minsky and Papert) Perceptron limitations The solution in the mid 1980's Multi-layer perceptron Back-propagation training

Summary of Applications Function approximation Pattern recognition/Classification Signal processing Modeling Control Machine learning

Biologically Inspired. Electro-chemical signals Threshold output firing Human brain: About 100 billion (1011) neurons and 100 trillion (1014) synapses

The Perceptron Sum of Weighted Inputs. Threshold activation function

Activation Function The sigmoid function: Logsig (Matlab)

Activation Function The tanH function: tansig (Matlab)

The multi layer perceptron (MLP) W1 W2 W3 f f f f f f zin ... f f f zout ... ... ... f f f 1 1 1 Y0 X1 Y1 X2 Y2 X3 Y3 W1 W2 W3 F1 F2 F3 1 zin zout

The multi layer perceptron (MLP) X1 Y1 X2 Y2 X3 Y3 W1 W2 W3 F1 F2 F3 1 zin zout

Supervised Learning Learning a function from supervised training data. A set of Input vectors Zin and corresponding desired output vectors Zout. The performance function

Supervised Learning Gradient descent backpropagation The Back Propagation Error Algorithm

BPE learning. f S X1 Y1 zin F zout W1 W2 f S 1 ... f S ... ... f S 1

Neural Networks 0 Collect data. 1 Create the network. 2 Configure the network. 3 Initialize the weights. 4 Train the network. 5 Validate the network. 6 Use the network.

Lack of information in the traning data. Collect data. Lack of information in the traning data. The main problem ! As few neurons in the hidden layer as posible. Only use the network in working points represented in the traningdata. Use validation and test data. Normalize inputs/targets to fall in the range [-1,1] or have zero mean and unity variance

Create the network. Configure the network. Initialize the weights. ... f S ... ... f S Only one hidden layer. 1 Number of neurons in the hidden layer

Train the network. Validate the network. Dividing the Data into three subsets. Training set (fx. 70%) Validation set (fx. 15%) Test set (fx. 15%) trainlm: Levenberg-Marquardt trainbr: Bayesian Regularization trainbfg: BFGS Quasi-Newton trainrp: Resilient Backpropagation trainscg: Scaled Conjugate Gradient traincgb: Conjugate Gradient with Powell/Beale Restarts traincgf: Fletcher-Powell Conjugate Gradient traincgp: Polak-Ribiére Conjugate Gradient trainoss: One Step Secant traingdx: Variable Learning Rate Gradient Descent traingdm: Gradient Descent with Momentom traingd: Gradient Descent Number of iterations.

Other types of Neural networks The RCE net: Only for classification. o X1 x o x o x x x o o x x x o o o x x x o o X2

Other types of Neural networks The RCE net: Only for classification. o X1 x o x o l x l x x S o o ... l x ... x x o o l o x x x o o X2

Parzen Estimator X Y G S G Xin Yout G / S G Yout x x x x x x x x Xin ... G / S ... G Yout x x x x x x x x Xin