NEURAL NETWORKS Biological analogy Introduction to Artificial Neural Networks Typical architectures

Biological neuron Soma: body of the neuron. Dendrites: receptors (inputs) of the neuron. Axon: output of neuron; connected to dendrites of other neurons via synapses. Synapses: transfer of information between neurons (electrical-chemical-electrical). João Sousa, José Borges XI.2

Neural networks Biological neural networks Artificial neural networks Neuron switching time: 0.001 second Number of neurons: 1010 Connections per neuron (synapses): 104,5 Recognition time: 0.1 s parallel computation Artificial neural networks Weighted connections amongst units Highly parallel, distributed process Emphasis on tuning weights automatically João Sousa, José Borges XI.3

Artificial Neural Networks Artificial Neuron Threshold function Piece-wise Linear Sigmoidal function João Sousa, José Borges XI.4

Use of Artificial Neural Networks Input is high-dimensional Output is multidimensional Mathematical form of system is unknown Interpretability of identified model is unimportant Applications Pattern recognition Classification Prediction Modeling Biological neural network Artificial neural network Soma Neuron Dendrite Input Axon Output Synapse Weight João Sousa, José Borges XI.5

Architectures of typical ANN Feedforward ANN João Sousa, José Borges XI.6

Architectures of typical ANN Recurrent ANN João Sousa, José Borges XI.7

ADAPTIVE NETWORKS Adaptive ANN Network Classification Backpropagation

Adaptive (neural) networks Massively connected computational units inspired by the working of the human brain Provide a mathematical model for biological neural networks (brains) Characteristics: learning from examples adaptive and fault tolerant robust for fulfilling complex tasks João Sousa, José Borges XI.9

Network classification Learning methods: supervised, unsupervised Architectures: feedforward, recurrent Output types: binary, continuous Node types: uniform, hybrid Implementations: software, hardware Connection weights: adjustable, hard-wired Inspirations: biological, psychological João Sousa, José Borges XI.10

Adaptive network Nodes can be static or parametric Network can consist of heterogeneous nodes Links do not have parameters associated Node functions are differentiable except at a finite number of points fixed nodes 3 x1 6 8 x8 4 7 9 x9 x2 5 adaptive nodes Input layer Layer 1 Layer 2 Output layer João Sousa, José Borges XI.11

Calculating with a network x g u y h v a x f y x a f y a y h v x g u João Sousa, José Borges XI.12

Backpropagation learning rule Simple gradient descent applied to layered networks An overall error measure is minimized for P data points and L layers Derivative information propagated by the use of chain rule, change in parameter a change in outputs of nodes containing a change in network's outputs change in error measure João Sousa, José Borges XI.13

Ordered vs. partial derivatives y x f g z partial derivative ordered derivative João Sousa, José Borges XI.14

BP for feedforward networks Define an error signal at each node output node hidden layer node João Sousa, José Borges XI.15

Error propagation network x1 x2 4 5 3 6 7 9 8 x8 x9 e1 e2 4 5 3 6 7 9 8 e8 e9 w83 w97 w52 w75 w31 João Sousa, José Borges XI.16